Exemplary embodiments of the present invention relate to a method and device for monitoring at least one vehicle passenger, as well as a method to control at least one assistance device.
German patent document DE 10 2009 053 407 A1 disclose a method for the continuous monitoring of the medical state of a vehicle passenger, in particular the vehicle driver in which at least one physiological signal of a physiological parameter of the vehicle passenger, a vehicle parameter and/or an environmental parameter are continuously detected, determined or analyzed. The physiological parameter of the vehicle passenger is issued continuously by means of an output unit at the side of the vehicle passenger, in particular continuously displayed on the instrument panel of the vehicle.
European patent document EP 1 182 089 B2 discloses a method for warning a driver of a vehicle in which a critical situation is detected by vehicle sensors. Furthermore, an alertness level of the driver is determined, wherein an image of the driver is captured by a camera device and the image is processed by a computing unit. An alertness level of the driver is determined from the image, wherein a line of vision of the driver is determined by the computing unit. In the case of a determined high alertness level of the driver, the issuing of the warning is dispensed with. In the case of several warnings to be issued at the same time, a first warning of a critical situation, which lies in the line of vision of the driver, is only issued after a second warning of a critical situation, which lies outside of the line of vision of the driver.
German patent document DE 10 2009 016 936 A1 assigned to the assignee of the present application, the complete content of which is hereby incorporated by reference, discloses a driver assistance system to support a driver of a vehicle in the case of fatigue. The driving assistance system comprises a drowsiness detection system to detect fatigue of the driver. In the case of fatigue of the driver recognized by means of the drowsiness detection system, a flashing warning light of the vehicle is activated to warn other road users.
Furthermore, in the article by Ming-Zher Poh, Daniel J. McDuff, and Rosalind W. Picard, “Noncontact, automated cardiac pulse measurements using video imagine and blind source separation,” Opt. Express 18, 10762-10774 (2010), the complete content of which is hereby incorporated by reference, discloses a method in which essentially the heart rate of the people is determined by means video sequences, captured by low-resolution video cameras under normal lighting conditions, of faces of people. The differences in brightness of the light reflected by the skin is measured and analyzed. First, the positions of the faces in the visual field of the camera are identified, then the video image in this section is broken up into red, green and blue portions and analyzed. During a heart beat the blood vessels—above all the arteries—expand slightly, as the pressure increases. This expansion causes an increase of the optical absorption and therefore also a decrease in the intensity of the light that is reflected by the face.
Exemplary embodiments of the present invention are directed to an improved method and device for monitoring at least one vehicle passenger in a vehicle and an improved method for controlling at least one assistance device of a vehicle.
In a method for monitoring at least one vehicle passenger in a vehicle, a current vital parameter value of at least one vital parameter of the vehicle passenger and, if necessary, a current environmental parameter value of at least one environmental parameter of an outer environment of the vehicle, are determined.
According to the invention, the vehicle passenger is identified and the current vital parameter value of the vehicle passenger is stored, if necessary together with the current environmental parameter value, and is compared to at least one older vital parameter value of the vehicle passenger, which has already been recorded and was recorded, if necessary, together with an older environmental parameter value, which corresponds to the current environmental parameter value. This means the current vital parameter value of the respective, clearly identified passenger is compared with one or several older vital parameter values of this vehicle passenger, if necessary with reference to the environmental context, wherein a short-term or long-term analysis of the vital state of the respective vehicle passenger is enabled by recording the vital parameter values. The comparison of the development of vital parameters over time comprises short-term, medium-term and longer-term changes. An analysis of progression over time of the vital parameter or a plurality of such vital parameters and furthermore preferably also parameters determined in another way, such as for example an alertness level, a line of vision, unconcentrated behavior, restlessness, fatigue and/or a stress level of the vehicle passenger, in particular of a respective vehicle driver, is enabled, if necessary with reference to the environmental context. This enables a vital state of the vehicle passenger, for example of a vehicle driver of the vehicle, to be assessed and suitable measures to be initiated in the case of a determined deterioration of the vital state, for example to activate functions of an assistance device of the vehicle, in order to at least indicate a potential risk of accident or, if necessary, to prevent an accident, to request help and/or to increase comfort of the vehicle passenger(s), in order to improve the vital state. This means the assistance device can, for example, be adapted to a current vehicle passenger state or to features of the vehicle driver, wherein his features can be evaluated by means of a stress level or the emotional state of the vehicle driver.
The clear identification of the respective vehicle passenger, the vital parameter values of whom are detected respectively, can, for example, occur by means of facial recognition by analyzing at least one currently captured image of the respective vehicle passenger or by another possibility of identification, for example by a registration in the vehicle by means of a password, by means of a code, by taking a finger print or by means of voice recognition. This means, for example, the people are clearly identified and the respective determined vital parameter values are allocated to the respective person and are stored and analyzed with reference to the person, even if several people use the vehicle interchangeably.
The vital parameters and/or symptoms that are able to be determined from these are monitored automatically over short, medium and/or longer time intervals and their trend development is observed. To that end, it is already sufficient if the respective vehicle passenger only spends a few minutes per day in the vehicle respectively.
In particular, a current state, in particular a state of vitality of the vehicle passenger can be very well assessed by the analysis of the current vital parameter value with reference to the environmental context in comparison to older vital parameter values, as, for example, in the new environmental, a deterioration of vital parameter values is normal due to a higher stress level. However, if the current vital parameter value deviates very strongly from the older vital parameter value, which was detected with a similar environmental parameter value, i.e. in a comparable environment, for example during a daily journey to work, a daily journey home, to the supermarket or the gym, i.e. during frequently reoccurring activities, or which was determined with very similar environmental parameters, for example at a road type or crossing type that is known to the vehicle driver, then this indicates a deterioration of the vital state of the driver and/or health problems, which could lead to a possible emergency situation or which can be counteracted by means of the assistance device, for example with a suitable comfort function of the vehicle. Thus, for example, pain relief can be effected by means of a massage function in a vehicle seat. In order to counteract an increased stress level or fatigue of the vehicle driver, essential oils can be sprayed by means of the ventilation and/or air conditioning function of the vehicle, which can be invigorating, refreshing or calming. For example, lavender oil is calming, thyme is invigorating, jasmine oil is soothing, calming and up-lifting and orange and lemon oil are mood-lifting. Furthermore, a chronic illness profile of the vehicle passenger can be reacted to accordingly, which can be determined by determination of (a) vital parameter(s), for example high blood pressure, a viral infection or a chronic obstructive pulmonary disorder (COPD).
This means an holistic situation analysis of the vehicle passenger, of the vehicle and of the vehicle environment is possible by means of the method and information and/or warnings to the vehicle passenger are to be formed dynamically and adapted to the determined and analyzed vital parameters. Furthermore, the assistance device(s) can be adapted to a vital state of the vehicle driver, in order to optimize in this way a security and a comfort of the vehicle passengers.
As they are recorded and stored in the vehicle, the stored vital parameters or vital parameter values are furthermore also available in an emergency situation for medical care. They can, for example, be transferred together with an automatically initiated emergency call to a hospital or casualty center, such that quick and adequate help can be initiated. Furthermore, for example, a long-term electrocardiogram can be produced and analyzed by a long-term analysis of the stored vital parameters and in this way, for example, a cardiac arrhythmia, also referred to as an arrhythmia, can be recognized and the vehicle passenger can be warned. For example, a visit to the doctor can be recommended to him/her.
The environmental parameter values can, for example, be determined by means of one or several environmental parameter sensor units, for example by means of a unit to determine the position of the vehicle by means of a global satellite navigation system to determine position. In this way, the determined vital parameter values can be allocated to a respective vehicle position and thus to a respective vehicle environment. Furthermore, alternatively or additionally, this environmental parameter sensor unit or a further environmental parameter sensor unit can be formed as a device to determine the environment of the vehicle, which has, for example, a radar, lidar and/or ultrasound sensor system and/or one or several cameras, for example mono and/or stereo cameras. In particular, for example, streets and/or crossings are detected by means of such cameras, such that the vital parameter values, for example, can be allocated to a respective street or crossing type, a respective traffic situation or a motorway situation. Thus the vital parameter values which were determined in connection with similar driving situations or vehicle environments can be compared with one another accordingly, such that, for example, short-term deteriorations of the vital parameters are very quickly able to be determined and long-term changes are able to be determined through the long-term analysis and can be indicated to the vehicle passenger.
Advantageously, a pulse rate and/or a breathing rate and/or a breathing volume and/or a heart rate variability and/or a skin temperature and/or a pumping capacity of the heart and/or a blood pressure and/or an oxygen concentration in the blood of the vehicle passenger, also referred to as oxygen saturation, and/or autonomous bodily functions, in particular functions of an autonomous nervous system of the vehicle passenger and/or pain statuses of the vehicle passenger is determined as a current vital parameter value. The heart rate variability (HRV) is connected to the heart rate and the breathing, i.e. the breathing rate and the breathing volume, via respiratory sinus arrhythmia, which describes a fluctuation of the heart rate in the synchronization with the breathing, such that it able to be determined via the determined heart rate, breathing rate and the determined breathing volume. Additionally vital functions or vital parameters or health risks such as resilience, stress tolerance, a cardio-vascular risk and further health risks, as well as positive and negative emotional states such as joy, stress or anger can be determined from the heart rate variability. For example, further diagnostic parameters are determined or recognized by means of the vital parameters, for example tachycardia, brachycardia and/or viral infections, by means of the heart rate variability, the breathing, i.e. the breathing rate and/or the breathing volume, and/or the skin temperature. The current vital state of the vehicle passenger can be sufficiently accurately assessed by means of these vital parameter values in order to be able to indicate, for example, an inattention of the vehicle passenger, for example of the vehicle driver, or a threatening or already occurring unconsciousness or severe health problems.
The current vital parameter value can, for example, be determined by means of a vital parameter sensor unit, which is arranged on a vehicle seat, on a seat belt or on a steering wheel or the vehicle. The sensor unit can be integrated into “smart textiles” in the vehicle or on the vehicle passenger.
For example, a plurality of such vital parameter sensor units for one or several vital parameters is also possible. In a particularly advantageous embodiment, the current vital parameter value, however, is determined, alternatively or additionally, by image analysis by means of at least one image of the vehicle passenger captured by an image capturing unit. This means the image capturing unit is the vital parameter sensor unit. In this way, the respective vehicle passenger, whose vital parameter values are determined, can also be clearly identified, for example by means of facial recognition, such that the determined vital parameter values can be clearly allocated to the respective vehicle passenger and can be stored and analyzed in the vehicle with reference to the person.
Therein, an image capturing unit that is already installed in the vehicle, for example in the region of a steering column or in the region of a visibly positioned telematic display, and is directed towards the vehicle passenger(s), preferably towards the vehicle driver, serves as an image capturing unit. Vital parameter values of several vital parameters of the respective vehicle passenger can also be detected simultaneously by means of this image capturing unit or all data can be detected simultaneously, from which the vital parameter values of the respective vehicle passengers can be determined by means of image analysis. Such image capturing units are installed in the vehicle, for example, to recognize alertness of the vehicle driver. Therein a video camera with low resolution, for example a so-called CCD camera, is already sufficient. Special lighting devices are also not required, as environmental lighting, for example daylight, is already sufficient. A robustness of a signal detection of the image capturing unit is clearly improved by partially redundant information, for example in a RGB color signal of the image capturing unit, and in particular by the possibility of the signal detection with a black and white camera for dark driving sections, i.e. for a dark environment, for example during a drive through a tunnel or at night. Therein, signal detection in the so-called HSC black and white signal occurs, if necessary by means of lighting by an infra-red light source. Therein, for example, a separate image capturing unit is not required, but the present image capturing unit is switched into a black and white mode. Advantageously, this occurs automatically depending on respective proportions of environmental light.
Preferably, the current vital parameter value is determined by detection of an optical absorption value and/or of an intensity value of reflected light of a skin region of the vehicle passenger and/or by a determination of a change of this optical absorption value and/or of this intensity value. During a heartbeat, blood vessels, in particular arteries, expand slightly, as pressure increases in the blood vessels. This expansion causes an increase of the optical absorption and therefore also a decrease of the intensity of the reflected light which is reflected by the skin region. By determining the differences in brightness of the light reflected by the skin region over time, vital parameter values, in particular the pulse rate, can be determined. This method is also called photoplethysmography. The foundations of the method are, for example, described in the article by Ming-Zher Poh, Daniel J. McDuff, and Rosalind W. Picard, “Noncontact, automated cardiac pulse measurements using video imaging and blind source separation,” Opt. Express 18 10762-10774 (2010), the complete contents of which is hereby included herein by reference.
Preferably, at least one region suitable for image analysis is searched for in each of the images during the image processing and only this region is analyzed. This region is a region in the images in which a sufficiently large skin region of the vehicle passenger is displayed. Preferably, a region of the images is searched for and analyzed as a region that is suitable for image analysis, which contains the face of the vehicle passenger, as in this region a sufficiently larger skin region that is able to be analyzed is available and as, in particular, image capturing units that are already installed in the vehicle and which are used, for example, to recognize alertness, are directed towards the face of the vehicle passenger, in particular of the vehicle driver. Vital parameter values of one or several vital parameters of several vehicle passengers can also be detected simultaneously with the method, wherein the image regions are determined, in which the faces of the vehicle passengers are situated and are analyzed separately from one another.
Advantageously, color images are captured as images of the vehicle passengers, in particular so-called RGB images, wherein the region of the images suitable for image analysis is split into individual color channels and at least one color channel is analyzed to determine the current vital parameter value. For example, the region suitable for image analysis is divided into red, green and blue portions, i.e. color channels. Therein, preferably, the green portion or color channel is analyzed, as the accuracy of the obtained results is highest with this color channel.
The skin temperature, the oxygen concentration or oxygen saturation and the blood pressure of the vehicle passenger, for example, can be determined from a combination of the color channels, more precisely from a phase shift and/or a change in amplitude between the color channels or color signals.
Preferably, a spectral power density of the color channel of the region suitable for image analysis is determined by means of a fast Fourier transform. The differences in brightness of the light reflected by the skin region over time can then be determined by means of the determined spectral power density and the vital parameter value(s) can be determined therefrom.
The vital parameter values determined in this way can also be combined with values of the vehicle passengers determined in another way, in order to thus obtain a better overall image of the state of the vehicle passenger. In this way, for example, fatigue or inattention of the vehicle driver can be determined in good time and with a low error rate. Thus, for example, movements of the respective vehicle passenger, in particular of the vehicle driver, can be determined by sensors integrated into the vehicle seat or into the vehicle seat textiles, which can indicate unconcentrated behavior, restlessness, stress or the beginnings of fatigue. This can, for example, be combined with the determined pulse rate, in order to obtain a better estimate.
A device to monitor at least one vehicle passenger in a vehicle comprises at least one vital parameter sensor unit to determine a current vital parameter value of at least one vital parameter of the vehicle passenger and at least on environmental parameter sensor unit to determine at least one current environmental parameter value of at least one environmental parameter of an outer environment of the vehicle.
According to the invention, the vehicle passenger is able to be identified and the current vital parameter value of the vehicle passenger is able to be stored in a storage unit together with the current environmental parameter value. The current vital parameter value is able to be compared in an evaluation unit with at least one older vital parameter value of the vehicle passenger, which has already been stored and which is stored together with an older environmental parameter value in the storage unit, which corresponds to the current environmental parameter value.
This enables a vital state of the vehicle passenger, for example a vehicle driver of the vehicle to be assessed and suitable measures to be initiated in the case of a determined deterioration of the vital state, for example functions of an assistance device of the vehicle to be activated in order to at least indicate a potential risk of accident or if necessary to prevent an accident, to request help and/or to increase comfort of the vehicle passenger(s) in order to thus improve the vital state, as has already been described with regard to the method, which is able to be carried out by means of the device, to monitor at least one vehicle passenger in a vehicle.
The clear identification of the respective vehicle passenger, the vital parameter values of which are detected respectively, can, for example, occur by means of facial recognition by analyzing at least one currently captured image of the respective vehicle passenger or by another possibility of identification, for example by a registration in the vehicle by means of a password, by means of a code, by taking a finger print or by means of voice recognition. This means, for example, the people are clearly identified and the respective determined vital parameter values are allocated to the respective person and stored and analyzed with reference to the person, even if several people use the vehicle interchangeably.
The environmental parameter sensor unit is, for example, a unit to determine the position of the vehicle by means of a global satellite navigation system for determining position. Furthermore, alternatively or additionally, this environmental parameter sensor unit or a further environmental parameter sensor unit can be formed as a device to detect the environment of the vehicle, which has, for example, a radar, lidar and/or ultrasound sensor system and/or one or several cameras, for example mono and/or stereo cameras. In particular, for example, roads and/or crossings are able to be detected by means of such cameras, such that the vital parameter values, for example, are to be allocated to a respective road and or crossing type or a respective traffic situation, for example a motorway situation. Thus, the vital parameter values, which were determined in connection with similar driving situations or driving environments are able to be compared with one another accordingly, such that, for example, short-term deteriorations of the vital parameters are able to be determined very quickly and long-term changes are able to be determined by the long-term analysis and can be indicated to the vehicle passenger.
The current vital parameter value is able to be compared with one or several older vital parameter values by means of the device with reference to environmental context, wherein a short-term, medium-term or long-term analysis of the vital state of the respective vehicle passenger is enabled by storing the vital parameter values. This means an analysis of progression over time of the vital parameter or a plurality of such vital parameters with reference to environmental context and preferably, furthermore, also parameters determined in a different such as for example an alertness level, a line of vision, unconcentrated behavior, restlessness, fatigue and/or a stress level of the vehicle passenger, in particular of a vehicle driver, is enabled.
The vital parameters and/or symptoms that are able to be determined from these are automatically able to be monitored over a longer period of time and their trend development is able to be observed. Therein it is already sufficient if the respective vehicle passenger only spends a few minutes per day in the vehicle respectively.
In particular, through the analysis of the current vital parameter value with reference to environmental context in comparison to older vital parameter values, a current state, in particular vitality of the vehicle passenger is able to assessed very well, as, for example, a deterioration of vital parameter values is normal in a new environment due to an increased stress level. However, if the current vital parameter value differs very strongly from an older vital parameter value, which was determined with the same environmental parameter value, i.e. in a comparable environment, for example during a daily journey to work, a daily journey home, to the shops or to the gym, i.e. during frequently reoccurring activities, or which was determined with very similar environmental parameters, for example at a road type or crossing type that is known to the vehicle driver, then this indicates a deterioration of the vital state of the driver or a health problem, which could lead to a possible emergency situation or which can be counteracted by means of the assistance device, for example with a suitable comfort function of the vehicle. Thus, for example, pain relief can be effected by means of a massage function in a vehicle seat. In order to counteract an increased stress level or fatigue of the vehicle driver, essential oils can be sprayed by means of the ventilation and/or air conditioning function of the vehicle, which can be invigorating, refreshing or calming. For example, lavender oil is calming, thyme is invigorating, jasmine oil is soothing, calming and uplifting and orange and lemon oil are mood-lifting. Furthermore, a chronic illness profile of the vehicle passenger can be reacted to accordingly which can be determined by determination of (a) vital parameter(s), for example high blood pressure, a viral infection or a chronic obstructive pulmonary disorder (COPD).
This means that an holistic situation analysis of the vehicle passenger, of the vehicle and of the vehicle environment is possible by means of the device and of the method that is able to be carried out with this, and information and/or warnings to the vehicle passenger are to be formed dynamically and are to be adapted to the determined and analyzed vital parameter. Furthermore, the assistance device(s) can be adapted to a vital state of the vehicle driver, in order to optimize in this way the security and comfort of the vehicle passengers.
Furthermore, the stored vital parameters or vital parameter values are available, for example, in an emergency situation for medical care, as they are recorded and stored in the vehicle. They can, for example be transferred together with an automatically initiated emergency call to a hospital or casualty center, such that quick and adequate help can be initiated. Furthermore, for example, a long-term electrocardiogram can be produced and analyzed by a long-term analysis of the stored vital parameters and in this way, for example, a cardiac arrhythmia, also called an arrhythmia, can be recognized and the vehicle passenger can be warned. For example, a visit to the doctor can be recommended to him.
The vital parameter sensor unit is, for example, arranged on a vehicle seat, on a seat belt or on a steering wheel of the vehicle. For example, a plurality of such vital parameter sensor units for one or several vital parameters is also possible. However, in a particularly advantageous embodiment, the device comprises, alternatively or additionally, at least one image capturing unit to capture images of the vehicle passenger and an image processing unit, by means of which a vital parameter value of the vital parameter of the vehicle passenger is able to be determined by image analysis of the captured image, as has already been described with respect to the method able to be carried out by the device.
In a method to control at least one assistance device of a vehicle using the method to monitor at least one vehicle passenger, according to the invention, at least one function of the assistance device is activated, if a current vital parameter value of at least one vital parameter of a respective vehicle passenger deviates from a predetermined value or value region for this vital parameter and/or deviates from a stored older vital parameter value of the respective vehicle passenger, which was stored together with an older environmental parameter value, which corresponds to the current environmental parameter value. In this way, the assistance device can intervene in a supporting manner in the case of poor or deteriorating vital parameter values of the vehicle passenger(s), in particular of the vehicle driver, in order indicate, for example, a potential risk of accident or if necessary to prevent an accident, to request help and/or to increase comfort of the vehicle passengers(s) in order to thus improve the vital state. This can occur very accurately as the vital parameter values are compared to one another with reference to environmental context and as the danger of a false evaluation of vital parameter values is greatly reduced in this way. Thus, for example, the functions of the assistance device are not activated, if a deterioration of vital parameter values is to be ascribed to an increased stress level due to an unknown environment, or for example, comfort functions are targetedly activated in order to thus enable a relaxation of the vehicle passenger. If the determined current vital parameter values deteriorates however with respect to older vital parameter values, which were detected in a comparable environment, for example on a daily journey to work, then this indicates an acute health problem and/or a deterioration of the vital state of the driver, such that corresponding other functions of the assistance device are to be activated.
Preferably, an intervention in a steering device, a braking device and/or a drive train of the vehicle and/or an optical, aural and/or haptic warning device is activated as a function of the assistance device. In this way, the vehicle, as, for example, is described in DE 10 2009 016 936 A1, the complete content of which is hereby incorporated by reference, can be steered automatically to the edge of a road and brought to a standstill there by a corresponding intervention in the steering device, braking device and/or drive train of the vehicle. Particularly advantageously, the intervention in the steering device, braking device and/or in the drive train of the vehicle can occur in such a way that the vehicle is steered autonomously to the nearest emergency assistance establishment, for example to the casualty of the nearest hospital.
Alternatively or additionally, warning devices of the vehicle can be activated, for example outer warning devices in the form of flashing warning lights and/or a horn of the vehicle, in order to warn other road users, and/or warning devices within the vehicle can be activated, in order to warn the vehicle passenger(s), for example in order to wake a vehicle driver who has fallen asleep or to indicate to other vehicle passengers a poor or deteriorating vital state of the vehicle driver such that they can operate the steering wheel and/or the handbrake of the vehicle in order to brake the vehicle and/or to steer the vehicle to the edge of the road.
Alternatively or additionally, advantageously at least one comfort function and/or emergency call function is activated as a function of the assistance device. Vital parameter values of the vehicle passenger(s) can be improved through the activation of the comfort function or a plurality of such comfort functions, for example if the determined vital parameter value(s) indicate fatigue, in particular of the vehicle driver. A massage function of a vehicle seat, interior ventilation and/or an air conditioning function of the vehicle can then be activated, for example. Therein, for example, essential oils or another refreshing medium can be introduced into the interior of the vehicle to refresh the vehicle passengers. Furthermore, the relevant vehicle passenger can be indicated about his current vital situation by optical, aural or haptic means, preferably with a recommendation of suitable counter measures, for example to take a restorative break, to move regularly and/or to take on sufficient fluids. In this way, the vehicle passenger can also be reminded, for example, to take required medication. Furthermore, devices of the vehicle to support or calm the driver can be activated in a targeted manner, for example in the case of an increased stress level recognized by means of the vital parameters, for example a navigation device or audio equipment of the vehicle, i.e. for example a car radio.
Additionally, the vehicle passenger(s) can also be warned of threatening health risks, for example of threatening cardio-vascular problems. Therein it is particularly advantageous to determine the vital parameters of the vehicle passenger(s), for example of the vehicle driver, over longer periods of time, for example several days or weeks and to analyze their development, in particular with reference to environmental context, such that slow deterioration can also be determined. In particular in the case of older passengers and/or passengers whose health is at risk or affected, in particular vehicle drivers, their sense of security and well-being is greatly increased. Additionally, in a medical emergency an automatic emergency call can be made, wherein preferably the detected vital parameter value can be transferred to an emergency call center or a hospital, such that quick and adequate medical help can be given. For this purpose, current positional date of the vehicle is preferably determined and transferred with the emergency call.
Parts that correspond to one another have the same reference numerals added to them in all figures.